Guides and indicators for eye movement monitoring systems

ABSTRACT

A computer system aids user positioning of a pointing device by providing an animated post warp retro guide in conjunction with an eye tracking system, based on perceived user intent, to assist a user in maintaining continuity for pointer warps. An eye tracking apparatus monitors a user&#39;s eye orientation while the user views a video display, detects the eye orientation relative to the video display, and automatically moves the positioning device to the calculated eye orientation of the user, while providing an animated history of the first location of pointing device and a second location of the pointing device. To enable the user to maintain continuity while reading lines of text, a reading guide and visual indicator are also provided. Automated text scrolling and paging is provided to further accommodate a user to maintain continuity while reading large blocks of text.

CROSS REFERENCE

This patent application is a divisional application of U.S. applicationSer. No. 10/727,237, filed Dec. 2, 2003 now U.S. Pat. No. 7,365,738. Thedisclosures of which is incorporated by reference in its entiretyinsofar as it does not conflict with the teachings of the presentinvention.

TECHNICAL FIELD

These teachings relate generally to the field of human-computerinteraction technology and more particularly to guides and indicatorsfor eye tracking systems.

BACKGROUND

Eye tracking technology determines the gaze point and length of gaze ofa user. It can indicate to a computer where on, and for how longthereupon, a user is looking at a computer monitor, thus replacing thetraditional PC mouse. Selections can be made by a slow blink or otherswitching mechanisms. Together with companion software, such ason-screen keyboards and voice recognition software, an eye trackersystem becomes a completely hands-free method for computer use.

A number of eye tracking devices are available that track the eyemovement and other elementary eye behaviors. Typical systems operate bymounting an eye tracking device adjacent to the computer monitor. Theeye monitoring device is generally equipped with an infraredlight-emitting diode, or LED. The LED bounces light off of the user'scornea and retina and the eye monitoring device records the reflectionsand sends the signals to software, which move the cursor to the spotwhere the user is looking. With these types of eye tracking systems, thecursor is positioned on a video screen according to the calculated gazeof the user. A number of different techniques have been developed toselect a target in these systems.

For example, Chang et al. disclose in U.S. Patent Application No.2002/0010707 A1, a system in which a user focuses on an annotation tagwhich can be determined by tracking the position of a conventionalscreen displayed pointer maneuverable by mouse, stylus, or keyboard, byuse of touch sensitive overlays to a screen, eye or head trackingsystem, or any other conventional mechanism for reliably determining auser focus on a region of a screen display. Initiating dynamic displayof supporting information at the user focus can be in response to userinitiated mouse clicks, keyboard input, finger taps, duration (e.g.mouse cursor held still on or adjacent to an annotation tag for onetenth of a second to about two seconds, with one second being typical),or any other suitable input scheme for confirming a user desire to viewsupporting bodies of data. Accommodation of the supporting body of datacan be through overlay or spatial morphing of the primary body of data.An overlay presentation replaces the annotation tag with supportingdata, while concurrently identifying the annotated region of the primarybody by suitable callouts, lines, markers, or animations that place theannotation in the margin, interline, or inter paragraph region, butstill direct a user's attention to the related region of the primarybody.

Tognazzini et al. disclose in U.S. Pat. No. 5,831,594, a method andapparatus for eye track derived backtrack to assist a computer user tofind a last gaze position prior to an interruption of eye contact. Thesystem of Tognazzini et al. indicates lost context by scrolling avirtual page and highlighting the last entity of a virtual page that hadthe last fixation immediately prior to the interruption. The lostcontext is marked in a number of ways to assist the user in reacquiringthe context.

Other conventional methods of assisting a user to recall screen contextinclude mouse trails and trackpoint accessibility features. A mousetrail is a feature of a Graphical User Interface (GUI) which causes themouse pointer to leave a trail of mouse arrows that fade away as a usermoves the mouse across the screen. Mouse trails are for normal usercontrolled mouse movements, and are a history of indicators of thesource of the mouse pointer. They typically appear on the screen for afew milliseconds. Mouse trails keep track of approximately eight recentpointer positions, only erasing the oldest. At any time, there may be upto, for example, eight sequential pointers on the screen, but if themouse is still, they will all be in the same position, and so only onewill be visible. When the mouse moves, it appears to leave a trail ofpointers behind it, and this can increase the visibility of the pointer.

A TrackPoint™ (TrackPoint is a registered trademark of InternationalBusiness Machines Corporation.) Accessibility feature is a pulsinganimation around the mouse pointer that is activated by the Control(Cntrl) key in Windows® XP (Windows is a registered trademark ofMicrosoft Corporation). It lets users find the mouse pointer more easilyby providing a visual cue via a brief animation to enable a user toquickly locate the pointer on the display screen. This option is helpfulfor people with visual impairments.

Other conventional solutions to maintain visual feedback for a user ofan eye tracking system include recording the screen location of where auser is looking and then jumping or “warping” the mouse pointer ahead tothat location. A “pointer warp” can be described as occurring when themouse pointer “jumps” to another part of the screen. A pointer warp isnot a smooth transition, instead it skips ahead. A problem with the useof the pointer warp is that it can be disconcerting as well asdisconnected. The mouse pointer jumps to the user's focus of attention,but there is no continuity with where it was previously. Another problemthat exists during use of the conventional pointer warp involves thereading of text as a user can lose track of the line that is beingcurrently read or when switching to a new line. As lines get longer, theproblem grows.

SUMMARY OF THE PREFERRED EMBODIMENTS

The foregoing and other problems are overcome, and other advantages arerealized, in accordance with the presently preferred embodiments ofthese teachings.

In one embodiment, the invention provides animated post-warp guides, inconjunction with an eye tracking system, to assist a user in maintainingcontinuity for pointer warps by reinforcing that the pointer has moved,and where it has moved from. This invention presents a quickly movingline to provide continuity to the user and to indicate where the userhas come from.

An alternate embodiment warps the mouse pointer and displays aconcentric pulse to subtly indicate the location of the mouse pointer.

A further embodiment of the invention assists readers of text. To avoidthe pointer getting in the user's way of reading online text, the eyetracking software recognizes the pattern created when the user begins toread and changes the pointer automatically to a “reading” guide thataids the user in the reading activity. The reading guide can assume anumber of suitable shapes and sizes.

A further embodiment of this invention automatically scrolls or pagesthe user through large blocks of text.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other aspects of these teachings are made more evidentin the following Detailed Description of the Preferred Embodiments, whenread in conjunction with the attached Drawing Figures, wherein:

FIG. 1 is a simplified block diagram of a computer system in which thepresent invention may be implemented;

FIG. 2A-2D is an illustration of an animated post pointer warp retroguide;

FIG. 3A-3B is an illustration of a concentric pulse indicator;

FIG. 4A illustrates a reading guide according to the present invention;

FIG. 4B illustrates the reading guide reverting to a pointer; and

FIG. 5 illustrates an alternate reading guide according to the presentinvention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The invention may be implemented as a software program stored on a datastorage device of a computer that operates in conjunction with anintegrated eye tracking/pointer positioning system, which may beembodied by various hardware components and interconnections asillustrated by the system 100 of FIG. 1. Generally, the system 100includes a computer 105, an eye tracking apparatus 110, a user inputdevice 115, and a display 120. The user input device 115 typicallycomprises a keyboard 117 and mouse 119, and an appropriate set ofsoftware drivers, shown as the user input software module 135.

The display 120 provides a medium for visually presenting text andgraphics to a user. The display 120 may be implemented by any suitablecomputer display with sufficient ability to depict graphical imagesincluding a cursor. For instance, the display 120 may employ a cathoderay tube, liquid crystal diode screen, light emitting diode screen, aprojected image, a holographic image, or another suitable videoapparatus. The images of the display 120 are determined by signals froma video module 125. The display 120 may also be referred to by othernames, such as video display, visual display, video screen, displayscreen, video monitor, display monitor, etc. The displayed cursor, alsoreferred to herein as a positioning device, visual pointer, mousepointer, or pointing device may comprise an arrow, cross-hair, or anyother image suitable for selecting targets or positioning an insertionpoint.

The computer 105 also includes one or more application programs 130,such as a web browser or a word processing program, an eye trackingmodule 140, which includes a software module configured to receive andinterpret signals from the eye tracking apparatus 110, and a centralprocessing unit (CPU) 145.

The system 100 is used by a “user”, also referred to as an “operator”.One skilled in the art will understand that many different types of eyetracking devices 110 can be used with the invention. For example,commercial eye gaze tracking systems are made by ISCAN Incorporated(Burlington Mass.), LC Technologies (Fairfax Va.), and Applied ScienceLaboratories (Bedford Mass.).

The eye tracking apparatus 110 is a device for monitoring the eye gazeor eye orientation of the computer user. The eye tracking apparatus 110may use many different techniques to monitor and detect the eyeorientation of the user, including, but not limited to cornealreflection and pupil tracking. One skilled in the art will recognize anumber of different devices suitable for use as the eye trackingapparatus 110.

The present invention introduces a method for eye tracking and automaticpointer warping. The invention automatically changes the mouse pointerbased on perceived user intent. The invention monitors the user's eyemovements, also referred to herein as the user's eye orientation or gazeposition, and if it appears that intent is there (i.e. the user fixatesfor a predetermined number of milliseconds on an area of the screen) theinvention immediately jumps the mouse pointer automatically to that spotof the visual display according to the calculated gaze of the user. Asan example, calculating the gaze of the user may be performed byreceiving x-y coordinate signals from the eye tracking module 140.Generally, conventional mouse trails do not provide useful informationto a user who is in direct control of the mouse pointer as the user isalready aware of the initial position of the mouse pointer as well asthe intended new location of the mouse pointer. Because the user of thepresent invention is less in direct control of the mouse pointer, mousehistory indication, as well as current location emphasis is useful tothe user for maintaining context during automatic pointer warping.

To help give the user continuity for pointer warps, reinforce that themouse pointer has moved, and where it has moved from, animated post-warpretro guides are provided as a visual indicator to the user. Referringto FIG. 2A, a basic illustration is depicted where a user is looking ata navigation link on a web site 205. The user in this example wishes topurchase a new monitor for his home computer. The user finds and looksat the monitor graphic 220 in the lower right of the computer displayscreen. The eye tracking apparatus 110 and software 140 sense that theuser is looking at that monitor graphic 220 and that he is interested init (e.g., the eyes stop moving and fixate at the monitor graphic for apredetermined number of milliseconds). Therefore, the mouse pointerautomatically warps to that spot representing the monitor graphic 220 ofthe display screen. Coinciding with this warp, the animated post warpretro guide 225 is displayed as illustrated in FIG. 2B, providing avisual indicator to indicate the initial position of the mouse pointer205 thus providing the user with continuity and a smooth transition.

The invention further helps indicate where the pointer currently is 230,referring to FIG. 2C and FIG. 2D. The movement of the animated post warpretro guide 225 enables the user to quickly locate the new warpedpointer location 230. The movement of the animated post warp retro guide225 also quickly enables the user to determine the initial location ofthe mouse pointer 205 which is a valuable feature as the movement of themouse pointer 205 is automated by the invention as contrasted to theconventional method of a mouse pointer being manually controlled by theuser. It can also be appreciated that the visual indicator provides aspatial relationship between the initial position of the mouse pointer205 and the new warped pointer location 230.

Referring to FIG. 3A and 3B, an alternate embodiment of the presentinvention warps the mouse pointer and shows a concentric pulse 240 tosubtly indicate that warping has occurred. The invention provides anenhanced environment for tracking mouse pointer movements usingconcentric pulses 240. In addition, the concentric pulse 240 may be usedin conjunction with the retro guide 225 if, for example, the user wishesto further accentuate the obviousness of the movement of the mousepointer. Conventionally, the mouse trail arrow pointer appears asmultiple instances along the movement line. The invention emphasizesnon-pointer ways to achieve the same result making it aestheticallypleasing to the user, and more indicative of past movement (retroguides) and current location (pulse), as opposed to multiple mousepointers along a line. The concentric pulse 240 is especially useful forother users following the pointer movement of another user (e.g., demos,class rooms, etc.).

Conventionally, a pulsating animation around the mouse pointer can bemanually activated by the user, for example, as in Windows® XP, bypressing the Control (Ctrl) key to assist a user in locating the mousepointer. This pulsating animation is beneficial for when the user losesthe mouse pointer in a corner, or otherwise can not locate the mousepointer. The present invention combines this pulsing animation withmouse pointer warping. So, while a conventional user must manually clickthe Ctrl key, the present invention automatically and seamlesslyprovides the location of the mouse pointer to the user as part of theoverall warping method. The pointer history, according to the presentinvention, is for warping automated mouse pointer movements, and alsofor emphasizing current pointer location via pulsing.

A further embodiment of the invention illustrates how the inventionassists readers of text. When in a reading mode, which is automaticallysensed by, for example, the user's left-to-right linear eye movementtied to the eye tracking software 140, the current line of text beingread is indicated with an open bracket in the margin to the left of theline 420, referring to FIG. 4A. Of course, the present invention can beadapted to sense other reading modes such as, for example, a right toleft linear eye movement. When the user slows down or stops on a link430, the “read mode” stops and the mouse pointer reappears 440 so theuser can click on the link, illustrated in FIG. 4B.

To avoid the pointer from getting in the user's way of reading onlinetext, the eye tracking software 140 recognizes the pattern created whenthe user begins to read and changes the pointer to a “reading” guide 420that aids the user in the reading activity. This guide 420 maintains aposition in the margin of the text, serving three purposes. First, thereading guide assists the reader in maintaining a fluid reading patternwithout losing the proper line location by marking the beginning of thecurrent line, and by automatically repositioning the reading guide atthe beginning of the next line of text as the user approaches the end ofthe current line of text. Second, by following the user's readingpattern, the guide 420 knows when to scroll the page when the userreaches the “fold”. Third, the guide 420 remains in the margin, ready torespond to the user's request for the mouse pointer, which is typicallyinitiated by a pause, or stare, which converts the guide 420 back to amouse pointer state 440 when needed. Referring to FIG. 5, the readingguide 420 can assume a number of suitable shapes and sizes.

A further embodiment of the invention automatically scrolls or pages theuser through large blocks of text. By following the user's readingpattern, the guide 420 knows when to scroll the page. As the user, whilein a reading mode, approaches completion of the viewable area of text(i.e., reaches the end of the text), the eye tracking software respondsby either gradually scrolling the text area ahead automatically for theuser or by abruptly paging ahead automatically for the user. Analternate embodiment of the invention for automatically scrolling theuser though large blocks of text maintains the user's current line oftext centered, or slightly off center vertically so the user has more ofa working margin above and below the current line, and can more easilyread ahead or back in the text.

While described in the context of eye track apparatus, any suitabletechnique for detecting and/or monitoring user's eye motion and/or eyeorientation (e.g. gaze direction) may be used including direct orindirect monitoring of visual signals.

Further, although described in terms of preferred embodiments, it shouldbe realized that a number of modifications of the teachings of thisinvention may occur to one skilled in the art and will still fall withinthe scope of this invention. By example, the present inventionspecifically makes reference to hardware relating to the automaticscroll control system that includes a computer system, display, screen,or monitor, other computer peripherals, and an eye tracking system.However, numerous other types of computer systems, displays, screens,monitors, and eye tracking systems, are well known to exist, and mostlikely, numerous other computer and tracking related systems, devices,and components will be developed in the future. The invention describedherein is not to be limited in use with specifically referenced types ofcomputer and tracking related systems, devices, components, andperipherals, but rather, are intended to be used with any and all typesof computer and tracking related systems, devices, components, andperipherals.

Further, while the teachings of this invention described herein areprovided with a certain degree of specificity, the present inventioncould be implemented with either greater or lesser specificity,depending on the needs of the user. Further, some of the features of thepresent invention could be used to advantage without the correspondinguse of other features. As such, the foregoing description should beconsidered as merely illustrative of the principles of the presentinvention, and not in limitation thereof.

1. A non-transitory computer readable medium having computer readableprogram code embodied therein for causing a computer to control theposition of a visual pointer using an eye tracking apparatus by:receiving input from the eye tracking apparatus; moving a visual pointerfrom a first location to a second location that corresponds to a user'sgaze position based on the input received from the eye trackingapparatus; providing a visual indicator between the first location andthe second location; automatically changing the visual indicator to areading guide in response to the eye tracking apparatus recognizing auser's gaze position pattern as a read mode, where the reading guide islocated in a margin at the beginning of a line of text that is read;repositioning the reading guide in response to the eye trackingapparatus determining that the user approaches the end of a line oftext; and in response to the eye tracking apparatus determining that theuser's gaze positions are one of slowing down or stopping on a link inthe text, exiting the read mode and changing the visual indicator to apointer for a pointing device to enable the user to click on the link.2. A computer readable medium as in claim 1, wherein the visualindicator comprises a substantially linear display element.
 3. Acomputer readable medium as in claim 1, wherein the visual indicatorcomprises a substantially circular display element.
 4. A computerreadable medium as in claim 1, wherein the visual indicator providesvisual continuity between the first location and the second location ofthe visual pointer.
 5. A computer readable medium as in claim 1, whereinthe visual indicator indicates the first location of the visual pointerand the second location of the visual pointer.
 6. A computer readablemedium as in claim 1, wherein the visual indicator provides a spatialrelationship between the first location of the visual pointer and thesecond location of the visual pointer.
 7. A computer readable medium asin claim 1, wherein the visual indicator comprises a graphic animationof a spatial relationship between the first location and the secondlocation of the visual pointer.
 8. A computer readable medium as inclaim 1, wherein moving the visual pointer to the second location isbased on inferring user intent from the user's detected gaze position.9. A computer readable medium as in claim 1, wherein the reading guidecomprises an open bracket.
 10. A computer readable medium as in claim 1,wherein the reading guide is positioned to the left of a line beingread.
 11. A computer readable medium program product as in claim 1,wherein the reading guide scrolls lines of displayed text in response tothe user's gaze position based on input received from the eye trackingapparatus.
 12. A computer readable medium as in claim 1, wherein thereading guide is changed to the pointer based on sensing a change in thegaze position of the user.
 13. A computer system comprising: aprocessor; a visual display output coupled to said processor; saidprocessor comprising an input configured to receive an input signal froman eye tracking apparatus, where the input signal indicates a user'sgaze positions, and said processor is further configured to provide adisplay signal at said visual display output for moving a visual pointerfrom a first location to a second location corresponding to the user'sgaze position, to generate a visual indicator between the first locationand the second location, to automatically change the visual indicator toa reading guide in response to the eye tracking apparatus recognizing auser's gaze position pattern as a read mode, where the reading guide islocated in a margin at the beginning of a line of text that is read, toreposition the reading guide in response to the eye tracking apparatusdetermining that the user approaches the end of a line of text, and inresponse to the eye tracking apparatus determining that the user's gazepositions are one of slowing down or stopping on a link in the text, toexit the read mode and change the visual indicator to a pointer for apointing device to enable the user to click on the link.
 14. A computersystem as in claim 13, wherein the visual indicator comprises asubstantially linear display element.
 15. A computer system as in claim13, wherein the visual indicator comprises a substantially circulardisplay element.
 16. A computer system as in claim 13, wherein thevisual indicator provides visual continuity between the first locationand the second location of the visual pointer.
 17. A computer system asin claim 13, wherein the visual indicator comprises a graphic animationof a spatial relationship between the first location and the secondlocation of the visual pointer.
 18. A computer system as in claim 13,wherein said processor is further configured to infer user intent fromthe user's detected gaze position and to move the visual pointer to thesecond location based on the inferred user intent from the user'sdetected gaze position.
 19. A computer system as in claim 13, whereinsaid processor is further configured to scrolls lines of displayed textin response to the user's gaze position based on input received from theeye tracking apparatus.
 20. A computer system as in claim 13, whereinsaid processor is further configured to change the reading guide to thepointer based on sensing a change in the gaze position of the user.